Model Selection in Regression

Variable Selection

Why might we not want to include all the variables available to us?

  • Overfitting: Using many extra variables gives the model more flexibility; it might be to tailored to the training data.
    • Recall: Polynomials in Week 1


  • Interpretability: We’d like to know which variables “matter most” to the response, and have accurate coefficient estimates.
    • What if two variables measure the same information?
    • What if the variables are linearly dependent?

Data

Recall: 62 unique variables describing Cannabis strains
New Response variable: Rating

cann <- read_csv("https://www.dropbox.com/s/s2a1uoiegitupjc/cannabis_full.csv?dl=1")

cann <- cann %>%
  dplyr::select(-Type, -Strain, -Effects, -Flavor, -Dry, -Mouth) %>% 
  drop_na(Rating)

Option 1: Best Subset Selection

Let’s try every possible subset of variables and pick the best one.

What do we mean by best?

Penalized metrics:

  • BIC
  • AIC
  • Mallow’s Cp
  • Adjusted R-squared

Where does cross-validation come in???

Option 1: Best Subset Selection

The problem:

Rating ~ Creative
Rating ~ Creative + Energetic
Rating ~ Creative + Tingly
Rating ~ Creative + Energetic + Tingly

\(\vdots\)

62 variables = \(2^{62}\) models = 4.6 quintillion models


Plus cross-validation????

Option 1: Best Subset Selection

If you have only a few variables, go for it.

In realistic settings, it’s not practical.

We’re going to try it anyway, using the leaps package.

Best Subset Selection with leaps

Best model of each size, based on R-squared:

library(leaps)
models <- regsubsets(Rating ~ Creative + Energetic + Tingly, 
                     data = cann, method = "exhaustive")

summary(models)
Subset selection object
Call: regsubsets.formula(Rating ~ Creative + Energetic + Tingly, data = cann, 
    method = "exhaustive")
3 Variables  (and intercept)
          Forced in Forced out
Creative      FALSE      FALSE
Energetic     FALSE      FALSE
Tingly        FALSE      FALSE
1 subsets of each size up to 3
Selection Algorithm: exhaustive
         Creative Energetic Tingly
1  ( 1 ) "*"      " "       " "   
2  ( 1 ) "*"      "*"       " "   
3  ( 1 ) "*"      "*"       "*"

Why can we automatically determine the best model within each size but we need to do another step to choose the best model across sizes?

Best Subset Selection with leaps

Now compare models:

                            Adj-R-Squared        CP        BIC
Creative                       0.01006460 16.694102  -8.840652
Creative, Energetic            0.01400814  8.454895 -11.303084
Creative, Energetic, Tingly    0.01633805  4.000000 -10.016876


What model would we choose?

Option 2: Backwards Selection

Start with all candidate variables in the model.

Drop the worst variable. (p-values or R-squared)

Check if dropping it helped. (penalized metric or cross-validation)

Stop when dropping is no longer good.

Backwards selection with leaps

models <- regsubsets(Rating ~ ., 
                     data = cann, 
                     method = "backward",
                     nvmax = 62)
Reordering variables and trying again:
summary(models)
Subset selection object
Call: regsubsets.formula(Rating ~ ., data = cann, method = "backward", 
    nvmax = 62)
62 Variables  (and intercept)
               Forced in Forced out
Creative           FALSE      FALSE
Energetic          FALSE      FALSE
Tingly             FALSE      FALSE
Euphoric           FALSE      FALSE
Relaxed            FALSE      FALSE
Aroused            FALSE      FALSE
Happy              FALSE      FALSE
Uplifted           FALSE      FALSE
Hungry             FALSE      FALSE
Talkative          FALSE      FALSE
Giggly             FALSE      FALSE
Focused            FALSE      FALSE
Sleepy             FALSE      FALSE
Earthy             FALSE      FALSE
Sweet              FALSE      FALSE
Citrus             FALSE      FALSE
Flowery            FALSE      FALSE
Violet             FALSE      FALSE
Diesel             FALSE      FALSE
`Spicy/Herbal`     FALSE      FALSE
Sage               FALSE      FALSE
Woody              FALSE      FALSE
Apricot            FALSE      FALSE
Grapefruit         FALSE      FALSE
Orange             FALSE      FALSE
Pungent            FALSE      FALSE
Grape              FALSE      FALSE
Pine               FALSE      FALSE
Skunk              FALSE      FALSE
Berry              FALSE      FALSE
Pepper             FALSE      FALSE
Menthol            FALSE      FALSE
Blue               FALSE      FALSE
Cheese             FALSE      FALSE
Chemical           FALSE      FALSE
Mango              FALSE      FALSE
Lemon              FALSE      FALSE
Peach              FALSE      FALSE
Vanilla            FALSE      FALSE
Nutty              FALSE      FALSE
Chestnut           FALSE      FALSE
Tea                FALSE      FALSE
Tobacco            FALSE      FALSE
Tropical           FALSE      FALSE
Strawberry         FALSE      FALSE
Blueberry          FALSE      FALSE
Mint               FALSE      FALSE
Apple              FALSE      FALSE
Honey              FALSE      FALSE
Lavender           FALSE      FALSE
Lime               FALSE      FALSE
Coffee             FALSE      FALSE
Ammonia            FALSE      FALSE
Minty              FALSE      FALSE
Tree               FALSE      FALSE
Butter             FALSE      FALSE
Pineapple          FALSE      FALSE
Tar                FALSE      FALSE
Rose               FALSE      FALSE
Plum               FALSE      FALSE
Pear               FALSE      FALSE
Fruit              FALSE      FALSE
1 subsets of each size up to 61
Selection Algorithm: backward
          Creative Energetic Tingly Euphoric Relaxed Aroused Happy Uplifted
1  ( 1 )  " "      " "       " "    " "      " "     " "     "*"   " "     
2  ( 1 )  " "      " "       " "    " "      "*"     " "     "*"   " "     
3  ( 1 )  " "      "*"       " "    " "      "*"     " "     "*"   " "     
4  ( 1 )  " "      "*"       " "    "*"      "*"     " "     "*"   " "     
5  ( 1 )  "*"      "*"       " "    "*"      "*"     " "     "*"   " "     
6  ( 1 )  "*"      "*"       " "    "*"      "*"     " "     "*"   "*"     
7  ( 1 )  "*"      "*"       " "    "*"      "*"     " "     "*"   "*"     
8  ( 1 )  "*"      "*"       " "    "*"      "*"     " "     "*"   "*"     
9  ( 1 )  "*"      "*"       " "    "*"      "*"     " "     "*"   "*"     
10  ( 1 ) "*"      "*"       "*"    "*"      "*"     " "     "*"   "*"     
11  ( 1 ) "*"      "*"       "*"    "*"      "*"     " "     "*"   "*"     
12  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
13  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
14  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
15  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
16  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
17  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
18  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
19  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
20  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
21  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
22  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
23  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
24  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
25  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
26  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
27  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
28  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
29  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
30  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
31  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
32  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
33  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
34  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
35  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
36  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
37  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
38  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
39  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
40  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
41  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
42  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
43  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
44  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
45  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
46  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
47  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
48  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
49  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
50  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
51  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
52  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
53  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
54  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
55  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
56  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
57  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
58  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
59  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
60  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
61  ( 1 ) "*"      "*"       "*"    "*"      "*"     "*"     "*"   "*"     
          Hungry Talkative Giggly Focused Sleepy Earthy Sweet Citrus Flowery
1  ( 1 )  " "    " "       " "    " "     " "    " "    " "   " "    " "    
2  ( 1 )  " "    " "       " "    " "     " "    " "    " "   " "    " "    
3  ( 1 )  " "    " "       " "    " "     " "    " "    " "   " "    " "    
4  ( 1 )  " "    " "       " "    " "     " "    " "    " "   " "    " "    
5  ( 1 )  " "    " "       " "    " "     " "    " "    " "   " "    " "    
6  ( 1 )  " "    " "       " "    " "     " "    " "    " "   " "    " "    
7  ( 1 )  " "    " "       " "    " "     "*"    " "    " "   " "    " "    
8  ( 1 )  " "    "*"       " "    " "     "*"    " "    " "   " "    " "    
9  ( 1 )  " "    "*"       " "    "*"     "*"    " "    " "   " "    " "    
10  ( 1 ) " "    "*"       " "    "*"     "*"    " "    " "   " "    " "    
11  ( 1 ) "*"    "*"       " "    "*"     "*"    " "    " "   " "    " "    
12  ( 1 ) "*"    "*"       " "    "*"     "*"    " "    " "   " "    " "    
13  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    " "   " "    " "    
14  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    " "   " "    " "    
15  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    " "   " "    " "    
16  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   " "    " "    
17  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   " "    " "    
18  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   " "    "*"    
19  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   " "    "*"    
20  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   " "    "*"    
21  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   " "    "*"    
22  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   "*"    "*"    
23  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   "*"    "*"    
24  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   "*"    "*"    
25  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   "*"    "*"    
26  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   "*"    "*"    
27  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   "*"    "*"    
28  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   "*"    "*"    
29  ( 1 ) "*"    "*"       "*"    "*"     "*"    " "    "*"   "*"    "*"    
30  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
31  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
32  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
33  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
34  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
35  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
36  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
37  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
38  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
39  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
40  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
41  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
42  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
43  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
44  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
45  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
46  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
47  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
48  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
49  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
50  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
51  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
52  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
53  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
54  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
55  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
56  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
57  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
58  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
59  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
60  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
61  ( 1 ) "*"    "*"       "*"    "*"     "*"    "*"    "*"   "*"    "*"    
          Violet Diesel `Spicy/Herbal` Sage Woody Apricot Grapefruit Orange
1  ( 1 )  " "    " "    " "            " "  " "   " "     " "        " "   
2  ( 1 )  " "    " "    " "            " "  " "   " "     " "        " "   
3  ( 1 )  " "    " "    " "            " "  " "   " "     " "        " "   
4  ( 1 )  " "    " "    " "            " "  " "   " "     " "        " "   
5  ( 1 )  " "    " "    " "            " "  " "   " "     " "        " "   
6  ( 1 )  " "    " "    " "            " "  " "   " "     " "        " "   
7  ( 1 )  " "    " "    " "            " "  " "   " "     " "        " "   
8  ( 1 )  " "    " "    " "            " "  " "   " "     " "        " "   
9  ( 1 )  " "    " "    " "            " "  " "   " "     " "        " "   
10  ( 1 ) " "    " "    " "            " "  " "   " "     " "        " "   
11  ( 1 ) " "    " "    " "            " "  " "   " "     " "        " "   
12  ( 1 ) " "    " "    " "            " "  " "   " "     " "        " "   
13  ( 1 ) " "    " "    " "            " "  " "   " "     " "        " "   
14  ( 1 ) " "    " "    " "            " "  " "   " "     " "        " "   
15  ( 1 ) " "    " "    " "            " "  " "   " "     " "        " "   
16  ( 1 ) " "    " "    " "            " "  " "   " "     " "        " "   
17  ( 1 ) " "    " "    "*"            " "  " "   " "     " "        " "   
18  ( 1 ) " "    " "    "*"            " "  " "   " "     " "        " "   
19  ( 1 ) " "    " "    "*"            " "  " "   " "     " "        " "   
20  ( 1 ) " "    " "    "*"            " "  " "   " "     " "        " "   
21  ( 1 ) " "    " "    "*"            " "  " "   " "     " "        " "   
22  ( 1 ) " "    " "    "*"            " "  " "   " "     " "        " "   
23  ( 1 ) " "    " "    "*"            " "  " "   " "     " "        " "   
24  ( 1 ) " "    " "    "*"            " "  " "   " "     " "        " "   
25  ( 1 ) " "    " "    "*"            " "  " "   " "     " "        " "   
26  ( 1 ) " "    "*"    "*"            " "  " "   " "     " "        " "   
27  ( 1 ) " "    "*"    "*"            " "  " "   " "     " "        " "   
28  ( 1 ) " "    "*"    "*"            " "  " "   " "     " "        " "   
29  ( 1 ) " "    "*"    "*"            " "  " "   " "     " "        " "   
30  ( 1 ) " "    "*"    "*"            " "  " "   " "     " "        " "   
31  ( 1 ) " "    "*"    "*"            " "  " "   " "     " "        " "   
32  ( 1 ) " "    "*"    "*"            " "  " "   " "     " "        " "   
33  ( 1 ) " "    "*"    "*"            " "  " "   " "     " "        " "   
34  ( 1 ) " "    "*"    "*"            " "  " "   " "     " "        " "   
35  ( 1 ) " "    "*"    "*"            " "  " "   " "     " "        "*"   
36  ( 1 ) " "    "*"    "*"            " "  " "   " "     " "        "*"   
37  ( 1 ) " "    "*"    "*"            "*"  " "   " "     " "        "*"   
38  ( 1 ) " "    "*"    "*"            "*"  " "   " "     " "        "*"   
39  ( 1 ) " "    "*"    "*"            "*"  " "   " "     " "        "*"   
40  ( 1 ) " "    "*"    "*"            "*"  " "   " "     " "        "*"   
41  ( 1 ) " "    "*"    "*"            "*"  "*"   " "     " "        "*"   
42  ( 1 ) " "    "*"    "*"            "*"  "*"   " "     " "        "*"   
43  ( 1 ) " "    "*"    "*"            "*"  "*"   " "     " "        "*"   
44  ( 1 ) " "    "*"    "*"            "*"  "*"   " "     " "        "*"   
45  ( 1 ) " "    "*"    "*"            "*"  "*"   " "     " "        "*"   
46  ( 1 ) " "    "*"    "*"            "*"  "*"   " "     " "        "*"   
47  ( 1 ) " "    "*"    "*"            "*"  "*"   " "     " "        "*"   
48  ( 1 ) "*"    "*"    "*"            "*"  "*"   " "     " "        "*"   
49  ( 1 ) "*"    "*"    "*"            "*"  "*"   " "     " "        "*"   
50  ( 1 ) "*"    "*"    "*"            "*"  "*"   " "     " "        "*"   
51  ( 1 ) "*"    "*"    "*"            "*"  "*"   " "     " "        "*"   
52  ( 1 ) "*"    "*"    "*"            "*"  "*"   " "     " "        "*"   
53  ( 1 ) "*"    "*"    "*"            "*"  "*"   " "     " "        "*"   
54  ( 1 ) "*"    "*"    "*"            "*"  "*"   "*"     " "        "*"   
55  ( 1 ) "*"    "*"    "*"            "*"  "*"   "*"     "*"        "*"   
56  ( 1 ) "*"    "*"    "*"            "*"  "*"   "*"     "*"        "*"   
57  ( 1 ) "*"    "*"    "*"            "*"  "*"   "*"     "*"        "*"   
58  ( 1 ) "*"    "*"    "*"            "*"  "*"   "*"     "*"        "*"   
59  ( 1 ) "*"    "*"    "*"            "*"  "*"   "*"     "*"        "*"   
60  ( 1 ) "*"    "*"    "*"            "*"  "*"   "*"     "*"        "*"   
61  ( 1 ) "*"    "*"    "*"            "*"  "*"   "*"     "*"        "*"   
          Pungent Grape Pine Skunk Berry Pepper Menthol Blue Cheese Chemical
1  ( 1 )  " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
2  ( 1 )  " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
3  ( 1 )  " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
4  ( 1 )  " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
5  ( 1 )  " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
6  ( 1 )  " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
7  ( 1 )  " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
8  ( 1 )  " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
9  ( 1 )  " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
10  ( 1 ) " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
11  ( 1 ) " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
12  ( 1 ) " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
13  ( 1 ) " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
14  ( 1 ) " "     " "   " "  " "   " "   " "    " "     " "  " "    " "     
15  ( 1 ) " "     " "   " "  " "   "*"   " "    " "     " "  " "    " "     
16  ( 1 ) " "     " "   " "  " "   "*"   " "    " "     " "  " "    " "     
17  ( 1 ) " "     " "   " "  " "   "*"   " "    " "     " "  " "    " "     
18  ( 1 ) " "     " "   " "  " "   "*"   " "    " "     " "  " "    " "     
19  ( 1 ) " "     " "   " "  " "   "*"   " "    "*"     " "  " "    " "     
20  ( 1 ) " "     " "   " "  " "   "*"   " "    "*"     " "  " "    " "     
21  ( 1 ) " "     " "   "*"  " "   "*"   " "    "*"     " "  " "    " "     
22  ( 1 ) " "     " "   "*"  " "   "*"   " "    "*"     " "  " "    " "     
23  ( 1 ) " "     " "   "*"  " "   "*"   " "    "*"     " "  " "    " "     
24  ( 1 ) "*"     " "   "*"  " "   "*"   " "    "*"     " "  " "    " "     
25  ( 1 ) "*"     " "   "*"  " "   "*"   " "    "*"     " "  " "    " "     
26  ( 1 ) "*"     " "   "*"  " "   "*"   " "    "*"     " "  " "    " "     
27  ( 1 ) "*"     " "   "*"  " "   "*"   " "    "*"     " "  " "    " "     
28  ( 1 ) "*"     "*"   "*"  " "   "*"   " "    "*"     " "  " "    " "     
29  ( 1 ) "*"     "*"   "*"  "*"   "*"   " "    "*"     " "  " "    " "     
30  ( 1 ) "*"     "*"   "*"  "*"   "*"   " "    "*"     " "  " "    " "     
31  ( 1 ) "*"     "*"   "*"  "*"   "*"   " "    "*"     " "  " "    " "     
32  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    " "     
33  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    " "     
34  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    " "     
35  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    " "     
36  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    " "     
37  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    " "     
38  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    " "     
39  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    " "     
40  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
41  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
42  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
43  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
44  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
45  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
46  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
47  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
48  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
49  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
50  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
51  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
52  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
53  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
54  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
55  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
56  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  " "    "*"     
57  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     " "  "*"    "*"     
58  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     "*"  "*"    "*"     
59  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     "*"  "*"    "*"     
60  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     "*"  "*"    "*"     
61  ( 1 ) "*"     "*"   "*"  "*"   "*"   "*"    "*"     "*"  "*"    "*"     
          Mango Lemon Peach Vanilla Nutty Chestnut Tea Tobacco Tropical
1  ( 1 )  " "   " "   " "   " "     " "   " "      " " " "     " "     
2  ( 1 )  " "   " "   " "   " "     " "   " "      " " " "     " "     
3  ( 1 )  " "   " "   " "   " "     " "   " "      " " " "     " "     
4  ( 1 )  " "   " "   " "   " "     " "   " "      " " " "     " "     
5  ( 1 )  " "   " "   " "   " "     " "   " "      " " " "     " "     
6  ( 1 )  " "   " "   " "   " "     " "   " "      " " " "     " "     
7  ( 1 )  " "   " "   " "   " "     " "   " "      " " " "     " "     
8  ( 1 )  " "   " "   " "   " "     " "   " "      " " " "     " "     
9  ( 1 )  " "   " "   " "   " "     " "   " "      " " " "     " "     
10  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
11  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
12  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
13  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
14  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
15  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
16  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
17  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
18  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
19  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
20  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
21  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
22  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
23  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
24  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     " "     
25  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     "*"     
26  ( 1 ) " "   " "   " "   " "     " "   " "      " " " "     "*"     
27  ( 1 ) " "   "*"   " "   " "     " "   " "      " " " "     "*"     
28  ( 1 ) " "   "*"   " "   " "     " "   " "      " " " "     "*"     
29  ( 1 ) " "   "*"   " "   " "     " "   " "      " " " "     "*"     
30  ( 1 ) " "   "*"   " "   " "     " "   " "      " " " "     "*"     
31  ( 1 ) " "   "*"   "*"   " "     " "   " "      " " " "     "*"     
32  ( 1 ) " "   "*"   "*"   " "     " "   " "      " " " "     "*"     
33  ( 1 ) " "   "*"   "*"   " "     " "   " "      " " " "     "*"     
34  ( 1 ) " "   "*"   "*"   " "     " "   " "      " " " "     "*"     
35  ( 1 ) " "   "*"   "*"   " "     " "   " "      " " " "     "*"     
36  ( 1 ) " "   "*"   "*"   " "     " "   " "      " " " "     "*"     
37  ( 1 ) " "   "*"   "*"   " "     " "   " "      " " " "     "*"     
38  ( 1 ) " "   "*"   "*"   " "     " "   "*"      " " " "     "*"     
39  ( 1 ) " "   "*"   "*"   " "     " "   "*"      " " " "     "*"     
40  ( 1 ) " "   "*"   "*"   " "     " "   "*"      " " " "     "*"     
41  ( 1 ) " "   "*"   "*"   " "     " "   "*"      " " " "     "*"     
42  ( 1 ) "*"   "*"   "*"   " "     " "   "*"      " " " "     "*"     
43  ( 1 ) "*"   "*"   "*"   " "     " "   "*"      " " " "     "*"     
44  ( 1 ) "*"   "*"   "*"   " "     " "   "*"      " " " "     "*"     
45  ( 1 ) "*"   "*"   "*"   "*"     " "   "*"      " " " "     "*"     
46  ( 1 ) "*"   "*"   "*"   "*"     " "   "*"      " " " "     "*"     
47  ( 1 ) "*"   "*"   "*"   "*"     " "   "*"      " " " "     "*"     
48  ( 1 ) "*"   "*"   "*"   "*"     " "   "*"      " " " "     "*"     
49  ( 1 ) "*"   "*"   "*"   "*"     " "   "*"      " " " "     "*"     
50  ( 1 ) "*"   "*"   "*"   "*"     " "   "*"      " " " "     "*"     
51  ( 1 ) "*"   "*"   "*"   "*"     " "   "*"      " " " "     "*"     
52  ( 1 ) "*"   "*"   "*"   "*"     " "   "*"      " " " "     "*"     
53  ( 1 ) "*"   "*"   "*"   "*"     "*"   "*"      " " " "     "*"     
54  ( 1 ) "*"   "*"   "*"   "*"     "*"   "*"      " " " "     "*"     
55  ( 1 ) "*"   "*"   "*"   "*"     "*"   "*"      " " " "     "*"     
56  ( 1 ) "*"   "*"   "*"   "*"     "*"   "*"      "*" " "     "*"     
57  ( 1 ) "*"   "*"   "*"   "*"     "*"   "*"      "*" " "     "*"     
58  ( 1 ) "*"   "*"   "*"   "*"     "*"   "*"      "*" " "     "*"     
59  ( 1 ) "*"   "*"   "*"   "*"     "*"   "*"      "*" " "     "*"     
60  ( 1 ) "*"   "*"   "*"   "*"     "*"   "*"      "*" " "     "*"     
61  ( 1 ) "*"   "*"   "*"   "*"     "*"   "*"      "*" "*"     "*"     
          Strawberry Blueberry Mint Apple Honey Lavender Lime Coffee Ammonia
1  ( 1 )  " "        " "       " "  " "   " "   " "      " "  " "    " "    
2  ( 1 )  " "        " "       " "  " "   " "   " "      " "  " "    " "    
3  ( 1 )  " "        " "       " "  " "   " "   " "      " "  " "    " "    
4  ( 1 )  " "        " "       " "  " "   " "   " "      " "  " "    " "    
5  ( 1 )  " "        " "       " "  " "   " "   " "      " "  " "    " "    
6  ( 1 )  " "        " "       " "  " "   " "   " "      " "  " "    " "    
7  ( 1 )  " "        " "       " "  " "   " "   " "      " "  " "    " "    
8  ( 1 )  " "        " "       " "  " "   " "   " "      " "  " "    " "    
9  ( 1 )  " "        " "       " "  " "   " "   " "      " "  " "    " "    
10  ( 1 ) " "        " "       " "  " "   " "   " "      " "  " "    " "    
11  ( 1 ) " "        " "       " "  " "   " "   " "      " "  " "    " "    
12  ( 1 ) " "        " "       " "  " "   " "   " "      " "  " "    " "    
13  ( 1 ) " "        " "       " "  " "   " "   " "      " "  " "    " "    
14  ( 1 ) " "        " "       " "  " "   " "   " "      " "  " "    " "    
15  ( 1 ) " "        " "       " "  " "   " "   " "      " "  " "    " "    
16  ( 1 ) " "        " "       " "  " "   " "   " "      " "  " "    " "    
17  ( 1 ) " "        " "       " "  " "   " "   " "      " "  " "    " "    
18  ( 1 ) " "        " "       " "  " "   " "   " "      " "  " "    " "    
19  ( 1 ) " "        " "       " "  " "   " "   " "      " "  " "    " "    
20  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  " "    " "    
21  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  " "    " "    
22  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  " "    " "    
23  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  "*"    " "    
24  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  "*"    " "    
25  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  "*"    " "    
26  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  "*"    " "    
27  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  "*"    " "    
28  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  "*"    " "    
29  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  "*"    " "    
30  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  "*"    " "    
31  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  "*"    " "    
32  ( 1 ) " "        " "       " "  " "   " "   " "      "*"  "*"    " "    
33  ( 1 ) " "        "*"       " "  " "   " "   " "      "*"  "*"    " "    
34  ( 1 ) " "        "*"       " "  " "   "*"   " "      "*"  "*"    " "    
35  ( 1 ) " "        "*"       " "  " "   "*"   " "      "*"  "*"    " "    
36  ( 1 ) " "        "*"       " "  " "   "*"   " "      "*"  "*"    "*"    
37  ( 1 ) " "        "*"       " "  " "   "*"   " "      "*"  "*"    "*"    
38  ( 1 ) " "        "*"       " "  " "   "*"   " "      "*"  "*"    "*"    
39  ( 1 ) " "        "*"       " "  " "   "*"   "*"      "*"  "*"    "*"    
40  ( 1 ) " "        "*"       " "  " "   "*"   "*"      "*"  "*"    "*"    
41  ( 1 ) " "        "*"       " "  " "   "*"   "*"      "*"  "*"    "*"    
42  ( 1 ) " "        "*"       " "  " "   "*"   "*"      "*"  "*"    "*"    
43  ( 1 ) "*"        "*"       " "  " "   "*"   "*"      "*"  "*"    "*"    
44  ( 1 ) "*"        "*"       " "  "*"   "*"   "*"      "*"  "*"    "*"    
45  ( 1 ) "*"        "*"       " "  "*"   "*"   "*"      "*"  "*"    "*"    
46  ( 1 ) "*"        "*"       " "  "*"   "*"   "*"      "*"  "*"    "*"    
47  ( 1 ) "*"        "*"       " "  "*"   "*"   "*"      "*"  "*"    "*"    
48  ( 1 ) "*"        "*"       " "  "*"   "*"   "*"      "*"  "*"    "*"    
49  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
50  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
51  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
52  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
53  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
54  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
55  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
56  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
57  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
58  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
59  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
60  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
61  ( 1 ) "*"        "*"       "*"  "*"   "*"   "*"      "*"  "*"    "*"    
          Minty Tree Fruit Butter Pineapple Tar Rose Plum Pear
1  ( 1 )  " "   " "  " "   " "    " "       " " " "  " "  " " 
2  ( 1 )  " "   " "  " "   " "    " "       " " " "  " "  " " 
3  ( 1 )  " "   " "  " "   " "    " "       " " " "  " "  " " 
4  ( 1 )  " "   " "  " "   " "    " "       " " " "  " "  " " 
5  ( 1 )  " "   " "  " "   " "    " "       " " " "  " "  " " 
6  ( 1 )  " "   " "  " "   " "    " "       " " " "  " "  " " 
7  ( 1 )  " "   " "  " "   " "    " "       " " " "  " "  " " 
8  ( 1 )  " "   " "  " "   " "    " "       " " " "  " "  " " 
9  ( 1 )  " "   " "  " "   " "    " "       " " " "  " "  " " 
10  ( 1 ) " "   " "  " "   " "    " "       " " " "  " "  " " 
11  ( 1 ) " "   " "  " "   " "    " "       " " " "  " "  " " 
12  ( 1 ) " "   " "  " "   " "    " "       " " " "  " "  " " 
13  ( 1 ) " "   " "  " "   " "    " "       " " " "  " "  " " 
14  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
15  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
16  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
17  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
18  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
19  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
20  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
21  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
22  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
23  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
24  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
25  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
26  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
27  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
28  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
29  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
30  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
31  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
32  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
33  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
34  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
35  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
36  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
37  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
38  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
39  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
40  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
41  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
42  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
43  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
44  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
45  ( 1 ) "*"   " "  " "   " "    " "       " " " "  " "  " " 
46  ( 1 ) "*"   "*"  " "   " "    " "       " " " "  " "  " " 
47  ( 1 ) "*"   "*"  " "   " "    " "       "*" " "  " "  " " 
48  ( 1 ) "*"   "*"  " "   " "    " "       "*" " "  " "  " " 
49  ( 1 ) "*"   "*"  " "   " "    " "       "*" " "  " "  " " 
50  ( 1 ) "*"   "*"  " "   "*"    " "       "*" " "  " "  " " 
51  ( 1 ) "*"   "*"  " "   "*"    " "       "*" "*"  " "  " " 
52  ( 1 ) "*"   "*"  " "   "*"    " "       "*" "*"  " "  "*" 
53  ( 1 ) "*"   "*"  " "   "*"    " "       "*" "*"  " "  "*" 
54  ( 1 ) "*"   "*"  " "   "*"    " "       "*" "*"  " "  "*" 
55  ( 1 ) "*"   "*"  " "   "*"    " "       "*" "*"  " "  "*" 
56  ( 1 ) "*"   "*"  " "   "*"    " "       "*" "*"  " "  "*" 
57  ( 1 ) "*"   "*"  " "   "*"    " "       "*" "*"  " "  "*" 
58  ( 1 ) "*"   "*"  " "   "*"    " "       "*" "*"  " "  "*" 
59  ( 1 ) "*"   "*"  " "   "*"    "*"       "*" "*"  " "  "*" 
60  ( 1 ) "*"   "*"  " "   "*"    "*"       "*" "*"  "*"  "*" 
61  ( 1 ) "*"   "*"  " "   "*"    "*"       "*" "*"  "*"  "*"

Backwards selection with leaps

                BIC
Model 13 -1159.2405
Model 14 -1155.1099
Model 15 -1150.6525
Model 16 -1144.7077
Model 17 -1138.9550
Model 18 -1133.3869
Model 19 -1127.5767
Model 20 -1121.7124
Model 21 -1115.7266
Model 22 -1109.1888
Model 23 -1102.7709
Model 24 -1096.3666
Model 25 -1089.9863
Model 26 -1083.7533
Model 27 -1077.4824
Model 28 -1071.4119
Model 29 -1065.5553
Model 30 -1060.1198
Model 12 -1057.7093
Model 31 -1054.8088
Model 32 -1049.5528
Model 33 -1043.8408
Model 34 -1037.9501
Model 35 -1032.0594
Model 36 -1026.2087
Model 37 -1020.2696
Model 38 -1014.3434
Model 39 -1008.1102
Model 40 -1001.7643
Model 41  -994.9492
Model 42  -988.3507
Model 43  -981.6881
Model 44  -974.8097
Model 45  -967.6551
Model 46  -960.5650
Model 11  -955.2681
Model 47  -953.3796
Model 48  -946.0997
Model 49  -938.7930
Model 50  -931.4421
Model 51  -924.0578
Model 52  -916.6737
Model 53  -909.2966
Model 54  -901.7930
Model 55  -894.2908
Model 56  -886.7031
Model 57  -879.0910
Model 58  -871.4425
Model 10  -869.8003
Model 59  -863.7226
Model 60  -855.9846
Model 61  -848.2431
Model 9   -789.0044
Model 8   -696.0323
Model 7   -614.3485
Model 6   -552.6699
Model 5   -494.0750
Model 4   -418.6211
Model 3   -348.0542
Model 2   -279.0113
Model 1   -186.0198

Backwards selection with leaps

Backwards selection with leaps

min_bic <- slice_min(model_bic, order_by = BIC) %>% 
  pull(model)

summary(models)$outmat[min_bic, ]
      Creative      Energetic         Tingly       Euphoric        Relaxed 
           "*"            "*"            "*"            "*"            "*" 
       Aroused          Happy       Uplifted         Hungry      Talkative 
           "*"            "*"            "*"            "*"            "*" 
        Giggly        Focused         Sleepy         Earthy          Sweet 
           "*"            "*"            "*"            " "            " " 
        Citrus        Flowery         Violet         Diesel `Spicy/Herbal` 
           " "            " "            " "            " "            " " 
          Sage          Woody        Apricot     Grapefruit         Orange 
           " "            " "            " "            " "            " " 
       Pungent          Grape           Pine          Skunk          Berry 
           " "            " "            " "            " "            " " 
        Pepper        Menthol           Blue         Cheese       Chemical 
           " "            " "            " "            " "            " " 
         Mango          Lemon          Peach        Vanilla          Nutty 
           " "            " "            " "            " "            " " 
      Chestnut            Tea        Tobacco       Tropical     Strawberry 
           " "            " "            " "            " "            " " 
     Blueberry           Mint          Apple          Honey       Lavender 
           " "            " "            " "            " "            " " 
          Lime         Coffee        Ammonia          Minty           Tree 
           " "            " "            " "            " "            " " 
         Fruit         Butter      Pineapple            Tar           Rose 
           " "            " "            " "            " "            " " 
          Plum           Pear 
           " "            " "

Option 1: Forward Selection

Start with one variable that you think is best.

Add the next best variable.

Test whether it was worth adding.

Keep going until it’s not worth adding any more variables.

Try it!

Open Activity-Variable-Selection

  1. Determine the best model via backwards selection.

  2. Fit that model to the data and report results.

  3. Determine the best model via forwards selection.

  4. Fit that model to the data and report results.